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ABSTRACT 



This is a guide for integrating data into the 
school- improvement process. It offers ways to use data to guide 
administrative and educational decision-making concerning instruction, 
curriculum, and programming. For those who are not statisticians, the guide 
can help in using and understanding data. It is based on the North Central 
Regional Educational Laboratory's Toolbelt Tutorial Web site. How to use data 
to define problems and needs, how to select improvement strategies and goals, 
how to initiate change, and how to evaluate students’ progress are explained. 
School- improvement processes are cyclical and continuous with four major 
activities: developing a plan, implementing the plan, evaluating the impact 
using specific criteria, and adjusting strategies to better meet criteria. 
These efforts are best taken collaboratively and reflectively. Educators must 
provide learning environments that are meaningful and engaging and then must 
reflect on the quality of their efforts, focusing on ways to make them 
better. Eight steps are taken in this process: (1) Leadership teams must be 

developed; (2) Data must be collected and organized; (3) Data must be 
analyzed for patterns; (4) Hypotheses must be posed for these patterns; (5) 
Then improvement goals must be set; (6) Specific strategies must be designed; 
(7) Evaluation criteria must be defined; and (8) the school must make a 
commitment to implement specific strategies, apply criteria to evaluate 
attainment of goals, and adjust strategies as necessary to better meet 
evaluation criteria. (Contains 4 references.) (RKJ) 
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This tutorial is designed to help educators incorporate data 
into their continuous school improvement process . 



Using Data to Bring About Positive Results 
in School Improvement Efforts 

North Central Regional Educational Laboratory 
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About This Tutorial 



Are you looking for ways to integrate data into the school improvement process? Would you like to find ways to use data 
to guide decision making about instruction, curriculum, programming, and so on? You are not alone. 

As the gap between low- and high-achieving students continues to grow and the implementation of high-stakes, 
performance-based accountability systems becomes the norm, the need for data, instead of intuition, philosophy, and 
retrospect, to guide administrative and educational decisions has become increasingly important. Unfortunately, many 
educators have little or no experience in using data systematically to inform decisions. They may have annual goals, but 
they have not been driven by the careful study of the school's and district's evidence of student learning. 

This tutorial is designed to help these educators incorporate data into their continuous school improvement process. And, 
since most educators are not statisticians, this tutorial offers some training on how to use and understand data. 

While this tutorial is by no means a comprehensive guide to data-driven decision making, it introduces eight steps 
educators can take to begin using data to define their problems and needs, select improvement strategies and goals, initiate 
change, and evaluate their students' progress. It is important to keep in mind that these steps were written to 
accommodate all of the states in the North Central region. Therefore, the steps are generic and not state specific. Each 
state may need to interpret and individualize these steps according to its needs. Regardless, it is our goal that after using 
this tutorial, educators will understand the importance of using data in the school improvement process and will begin to 
apply some of the recommendations. 



Introduction 



As educators take the journey to bring their students to standards that exemplify world-class achievement, they find 
themselves embarking on new territory. The journey can be somewhat confusing and frustrating, or it can be clear and 
rewarding. What we do know is that the adventure is best undertaken collaboratively and reflectively. 

Collaboration asks members of a school community to join in ongoing problem-solving ventures, pooling then- 
knowledge, talents, and ideas. In school systems, district and building leaders join teachers, support staff, and parents in 
teams to explore improvement issues. Easier said than done, successful collaboration requires leadership skills in creating 
numerous and diverse partnerships, sustaining a vision, group problem solving, conflict resolution, and compromise. 

Reflection , a companion to collaboration, asks teams to think about the information in front of them and to adjust then- 
actions accordingly. On the school improvement journey, reflection is necessary not only to stay on the improvement path 
but to discover the best path. Successful reflection depends on thought-provoking information and time for individual and 
team study. 

Reflective collaboration is a powerful process that occurs among team members. When we are reflecting about improving 
student achievement, the necessary information can be found in our system's data. The data in our schools provide 
important clues about our work and our students' performance. But how do we embark on this reflective collaboration 
process? 

First, it requires time — time during the day and the week to involve teachers, always a challenge. We have found that 
schools committed to using data to guide their work allocate time for teachers to meet, discuss, reflect upon data, and 
make informed instructional decisions. Schools identify the need for this time, then find it through a combination of 
creative scheduling (e.g., having all first-grade teachers share student data while students attend "specials" such as art and 
music), and priority setting (using weekly faculty meetings to analyze student data). 

Second, continual exposure to data will help to build a district and school culture that values the use of reliable, complete 
information to guide decisions and solve problems. We understand that for many people the idea of working with data is 
unfamiliar and perhaps uncomfortable. The fact is, whether we realize it or not, we use data every day to help us make 
decisions. We listen to weather updates, look over stock market reports, and read healthy-living tips in magazines. Just 
as data helped you make personal decisions, they will help your team make decisions about school improvement. 
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Improving Student Learning 



The underlying assumption for school improvement efforts is that student learning can and should improve on a 
continuous basis. Students come to our schools to learn, to find exciting challenges and new understandings. If we are to 
provide learning environments that are meaningful and engaging, we must continually reflect on the quality of our 
systems and focus our efforts to make them better. The section on the School Improvement Cycle describes how and 
why data can and should be incorporated into this continuous improvement process. In addition, the section on Successes 
of Data-Driven Decision Making provides Web site addresses to schools and districts that have used and succeeded in 
incorporating data into their school improvement cycle. These district and school leaders are guided by a clear vision 
focused on student learning and by a well-defined mission statement aimed at high-quality learning environments 
resulting in optimum student achievement. These insightful leaders empower collaborative teams, engage their staff in 
purposeful analysis of their systems, and guide them through data-driven decision making. 



J 
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The School Improvement Cycle 



The School 
Improvement Cycle 




Effective school improvement processes are cyclical and continuous, 
with no clear beginning or end. The school improvement cycle 
shown to the left was developed by Dr. Walter Shewhart and 
provided a foundation for much of the work of W. Edwards Deming 
(see Rinehart, 1993). 

This cycle contains four major activities: 



• Plan: Develop a plan for improvement 

• Do: Implement the plan. 

• Study: Evaluate the impact according to specific 
criteria. 

• Act: Adjust strategies to better meet criteria. 



We realize that, in spite of our good intentions, not every intervention will be successful for every child, and at times our 
efforts may not lead to the results we had anticipated. But with rigorous measurement of our work, informed decision 
making, and a willingness to change, the improvement process can be a forgiving one. That is, when we evaluate how 
interventions, such as using new teaching techniques, affect student learning, we learn what, and for whom, they are 
working. With this information, we adjust our practices, renew our plans, and try again. We work to continuously 
improve. 



Data are the key to continuous improvement. When we "Plan," we must use data to provide insight and focus for our 
goals. Data patterns reveal strengths and weaknesses in the system and provide excellent direction. When we "Do," we 
collect data that will tell us the impact of our strategies. Through collaborative reflection, we "Study" the feedback offered 
by our data and begin to understand when to stay the course and when to make changes. Then we "Act" to refine our 
strategies. 

As shown in the table below, focusing on data throughout the school improvement cycle, rather than on intuition, 
tradition, or convenience, marks a great change in what administrators and teachers have used in the past to drive their 
decision making regarding student learning. 
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Decision Making Based on Intuition, 
Tradition, or Convenience 


Data-Driven Decision Making 


• Scattered staff development programs 


• Focused staff development programs 
as an improvement strategy to address 
documented problems/needs 


• Budgetary decisions based on prior 
practice, priority programs 


• Budget allocations to programs based 
on data-informed needs 


• Staff assignments based on interest 
and availability 


• Staff assignments based on skills 
needed as indicated by the data 


• Reports to the community about 
school events 


• Organized factual reports to the 
community about the learning 
progress of students 


• Goal-setting by board members, 
administrators, or teachers based on 
votes, favorite initiatives, or fads 


• Goal-setting based on data about 
problems and possible explanations 


• Staff meetings that focus on 

operations and the dissemination of 
information 


• Staff meetings that focus on strategies 
and issues raised by the local school's 
data 


• Parent communication via twice-a- 
year conferences at elementary "open 
houses" and newsletters 


• Regular parent communication 

regarding the progress of their children 


• Grading systems based on each 
teacher's criteria of completed work 
and participation 


• Grading systems based on common 
criteria for student performance that 
reports progress on the standards as 
well as work skills 


• Periodic administrative team meetings 
focused solely on operations 


• Administrative team meetings that 
focus on measured progress toward 
data-based improvement goals 



Successes of Data-Driven Decision Making 

Following are links to sites that describe how schools and districts have found ways to make data-driven decision making 
work for them: 



• Teachers and Students as Action Researchers: Using Data Daily. Creve Coeur, IL 
(NCREL) http://www.ncrel.org/info/nlp/lpsuOO/creve.htm Especially valuable for those 
interested in getting students involved in the data-driven decision making process. 

• Continuous Improvement: Monitoring the quality of student learning. Ashatabula, OH 
(NSDC) http://www.nsdc.org/library/tools/2-981ead.html Another example of getting 
students involved in a data-driven school improvement process. 

• Data-driven Improvement Effort Leads to Results in Oak Park. Oak Park, MI (NSDC) 
http://www.nsdc.org/library/results/9-97rich.html This school involved parents and 
community members in the improvement process and automated some aspects of data 
collection. 

• New routes open when one type of data crosses another. Northern CA (NSDC) 
http://www.nsdc.org/library/isd/bemhardt2 1 1 .htrnl Background on using data in schools, 
plus a case study crossing demographic data with results on third-grade reading 
assessments. 
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The Steps 



How should districts and schools begin to incorporate data into their school improvement plans? The North Central 
Regional Educational Laboratory recommends eight steps that have been successfully used in the Data Retreats sponsored 
by the CESA 7 Standards and Assessment Center in Green Bay, Wisconsin: 

1 . Develop a Leadership Team 

2. Collect and Organize Data 

3. Analyze Data Patterns to Define Problems/Needs 

4. Pose Hypotheses 

5. Set Improvement Goals 

6. Identify Specific Strategies 

7. Define Evaluation Criteria 

8. Make the Commitment to: 



• Implement Specific Strategies. 

• Apply Criteria to Evaluate Attainment of Goals. 

• Adjust Strategies as Necessary to Better Meet Evaluation Criteria. 

Teams that are new to data-driven school improvement will focus much of their initial efforts toward "Planning." As 
such, many of these steps, explained now in detail, are critical to that activity. Future versions of this tutorial will expand 
upon work that is done as we "Do," "Study," and "Act." 



Step 1: Develop a Leadership Team 

In order for data to be successfully incorporated into the school improvement cycle, school and district representatives 
must form a team. A team (rather than an individual or small group) is ideally suited for this work because: 

• The steps to incorporate data into the school improvement cycle take a lot of work and require the 
commitment of many individuals. 

• The data that can be used come from a variety of sources. It is important to have representatives 
with different perspectives to ensure that various sources of vital data are not overlooked. 

• Discussions are richer and more diverse with numerous points of view and insights. 

• Dissemination of information is much easier when there are multiple people who can remember and 
share experiences. 

• The effort needed to sustain continuous school improvement during the current and subsequent 
school years is much easier when tasks are divided among a team of people. 

For these reasons, district leaders must work to develop leadership teams that include members from the school and from 
the wider community (parents, business leaders, and others with an interest in the school). Members from both the 
school and the community must reflect the schools' student population in terms of racial/ethnic and special needs. 
Sometimes called "improvement teams," or "learning teams," these members should come together regularly (at a 
minimum, once a month) to discuss and plan efforts to improve student learning for all students. The size of the team 
may vary with the size of the district. 

Team Membership 

We recommend that district leadership teams include members as listed in the table below. All meetings should be 
mandatory and should include representation from every building principal, the special education coordinator, the 
curriculum coordinator, and the superintendent. The number of teachers may be kept as single representatives during the 
school year, but additional teachers and other members may be added depending on programs and the configuration of 
the district. These teams should be kept to a manageable size. When teams become too large, their meetings are less likely 
to achieve progress during the school year. Our observations are that teams of 15 or fewer people can be effective. 



Makeup of District Leadership Teams 


District Superintendent 


3-5 Elementary Teacher Representative(s) 


All Building Principals 


Middle School Teacher Representative(s) - Core 
subjects 


Special Education District Representative 


High School Teacher Representative(s) - Core subjects 


Curriculum District Representative 


Noncore Subjects Teacher Representative(s) 


Special Programs Representative (Title I, At Risk, 
Gifted, and Talented, etc.) 


Parent Representative(s): preferably hard to reach 
parents and parents who are not employees of the 
school district 


Guidance Counselors Representative and/or Other 
Pupil Services Staff 


School Board Member 


Assessment District Representative 


Business/Community Representative(s) 


PK-12 Teacher Representative(s) 





Team Qualities 

The two qualities each individual on the leadership team should have are collaborative skills and an appreciation for data. 
The school improvement cycle is not a book that can be studied and interpreted. Rather it is a group process that requires 
a team of educators who collaborate and who all understand the value in collecting, analyzing, and using relevant data to 
guide decision making. 

The more time that members spend working together, the more they will build their collaborative skills and team spirit. 
The team facilitator should set the tone by modeling and teaching valuable team skills. The team leader should also 
encourage members to make firm commitments to the cause: using data to improve student learning. 
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Step 2: Collect and Organize Data 



Collecting the data should be a planned, purposeful process. Valuable 
data will guide the school improvement team in developing 
improvement goals for the benefit of all students. There are four types 
of collectable data to use as indicators of school or district success and 
progress. The initial and central focus point is on assessment data, data 
that indicate what students know. Three other valuable types of 
data — student, program, and perceptions — may contribute and give 
insight to assessment results. 

Prior to the school year, the administrative team should review and 
select from available sources of data. To do this successfully, the team 
needs to develop a plan that will set forth processes to collect important 
data throughout the school year. This data collection plan should be a 
purposeful blueprint for gathering key descriptive information. The 
following four sections provide information and guiding questions that 
own blueprint for collecting the four types of data. 




are crucial for teams to use when designing their 



1. Assessment Data 

As indicated above, the most important type of data to focus on is student assessment data. A comprehensive assessment 
plan makes use of data from three tiers. These tiers vary according to their purposes, rate and type of feedback they 
provide, and their targeted audience. This information is summarized in the table below: 









Tier 


Assessment 


Rate of 


Type of 


Primary Target 








Purpose 


Feedback 


Feedback 


of Feedback 
















General, 


General 












Infrequent 


broad 


accountability 


p 

<-■: 




III 


Annual 

large-scale 


i 


i 


i 


i 


audience: 

Policymakers 

Community 

Administrators 
























Others 




m 


m. 




II 


Periodic 

grade 










Administrators 








level/subject 










Teachers 










area 


















I 


Ongoing 










Teachers 








classroom 


▼ 


▼ 


Students 












Frequent 


Specific, 


















narrow 





Tier III: Annual Assessment Data 

Tier III data, such as annual state assessments, is designed primarily for accountability purposes-to report to external 
members of the school community a broad view of the district's achievement levels. Its primary school community 
audience is composed of board members, administrators, and program leaders. Tier III data also can be useful to 
curriculum teams that use the information to evaluate the general effectiveness of the curriculum, and by secondary school 
community members (teachers, students, and parents). 

State assessments have limited use because they are designed to sample broad domains of student knowledge. They are 
administered once a year and can be used as broad indicators of the school's effectiveness. 

It is not surprising that team members become frustrated when they analyze their Tier III assessment data — they can take 
it just so far. While the assessments can provide valuable information about the district's general success, it is not helpful 



when evaluating student progress, and it does not provide useful data during the school year. 

Tier III assessments cannot: 

• Help a teacher adjust lesson plans during the school year. 

• Help teams make placement or program decisions during the school year. 

• Provide information on a student's progress during the school year. 

• Provide more detailed information about the student's skill attainment toward the standard. 

• Show a student's depth of conceptual understanding. 

Tier II: Periodic Assessment Data 

Throughout the school year, periodic assessments efficiently provide immediate results of student performance on key 
standards-based skills in a content area and grade level. Periodic assessments can be used to: 

• Establish the entrance-level performances of students when the school year begins. 

• Indicate progress with strengths and weaknesses in a particular content area in the middle of the 
year. 

• Create groupings of students based on their changing skill needs. 

• Identify which students need enrichment or which students need special assistance at any point 
during the school year. 

• Help document the success of school programs. 

To plan periodic assessments, a team can use the following steps as a guide: 

• Discuss the core subject areas and decide when periodic assessments might be useful. Which 
skills and understandings should be assessed? 

• List the purposes for and the users of periodic assessments in these subject areas. 

• Discuss the parameters of the assessments. How long should they be? What criteria should be 
used to judge their integrity? 

• What types of tasks would you pose to the students? 

• How often would the assessments be administered? 

• How should the assessments be administered? by paper and pencil booklets? by booklets with 
scannable score sheets? by computer? 

When these questions are answered, the team will need to find a source for the assessments. Can teachers develop the 
assessments? Can the assessments be obtained by a test publisher or vendor? Can textbook or curriculum materials 
provide a source? Factors such as cost, proven reliability, validity for intended purposes, types of reported scores, and 
scheduling all influence the decision. The table below provides a list of several examples of periodic assessments. 



Assessment 


How Often 


Type of Data 


How Used 


Leveled Reading 
Assessments of 
fluency and 
comprehension, 
Grades 1-2 


4 times a 
year 


Level designation 


To provide reading instructional 
support for students who are not 
progressing; to provide students guided 
reading at their designated levels 


Measures of 
Academic Progress 
(MAP) Leveled Tests 
aligned to local 
standards in math, 
reading or language 
by NWEA 
(Northwest 
Evaluation 
Association) in 
Grades 2-10 


3 to 4 times 
a year 


An individualized Rasch 
Index score can be used to 
measure individual student 
growth 


To measure progress and track 
individual student growth over time 


6 + 1 Trait Writing 
Prompts at Grades 
2-12 


3 times a 
year 


Rubric scores 


To guide writing instruction in the traits 



0 

ERJC 



These assessments, if designed well, actually become embedded within instruction. Well-designed assessments tell 
teachers what does and does not work for more effective instruction. 
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Crucial to the success of periodic assessments is their use. Training and support should be provided so that teachers can 
study the results and make immediate decisions about their teaching. The assessments should provide clear indications of 
learning progress. Some teachers find the results most helpful when they can share them at grade-level or subject- area 
team meetings and when they can get group ideas for how to assist students at risk. 

Periodic assessment data should be collected and used during the school year so that it eventually can be incorporated into 
the school improvement cycle. Tier II periodic assessment information will change the reflective collaboration that team 
members engage in at leadership team meetings. Teams will discuss progress, consequences, and actions, focusing much 
more on what kind of action was taken for students with specific needs. Team members and all staff will be assuming 
more responsibility for students who soar academically and for students who struggle. 

Tier I: Ongoing Classroom Assessment Data 

At Tier I lies the heart of assessment, the classroom. Building a culture of assessment means making assessing and using 
data a natural part of every teacher’s professional repertoire. Assessing performance on the standards means assessing the 
depth of conceptual understanding as well as knowledge and skills. There is an entire continuum of assessment options 
that meet a variety of purposes. But even if teachers are implementing a variety of assessment methods, they are 
meaningless unless their results are USED to make decisions. 

We must challenge ourselves to lay out the data from daily assessments in a way that shows clearly who is excelling and 
needing enrichment, who is performing on target, and who needs help. Then, it is our challenge to find a way to provide 
that help. Our grade books must be used more meaningfully, not merely for assigning grades but for charting useful 
information in making decisions. Every lesson and unit plan should be devised based on assessment data. 

All decisions regarding extra tutorial help, enrollment in course sequences, materials to use, and groupings of students 
should be flexible based on what the data reveal about student learning. Students may excel in one skill but stumble on 
the next. Our lessons should be focused on the standards but allow for adaptability based on real student performance. 

When meaningful data are kept and used in grade books, charted and displayed with students, or organized in portfolios 
or electronic decision support systems, teachers, and often students, are "in the know." Based on these data, decisions 
can be founded solidly on how students are performing. 

Assessments as Snapshots 

Assessment data are sometimes referred to as "snapshots" of student knowledge. The snapshot metaphor is useful for 
considering the values of and limitations related to different tiers and different types of assessment data. Consider the 
following: 



Snapshots 
reflect 
how the 
subject 
looks at a 
specific 
point in 
time. 




• Similarly, 
assessment data 
(from large-scale 
to 

classroom-based) 
aim to represent 
a student's or a 
group of 
students' 
knowledge at the 
time of testing. 






• A series of snapshots, taken over time, may reveal changes in the subject. 




• Similarly, it is only with a series of assessment data, sampled over time, that changes in student 
knowledge, or learning, may be ascertained. This is true for each tier of assessment. 




• Multiple snapshots, taken from 
different angles and with 
different lenses, may reveal 
additional information about 
the subject 




• Similarly, data resulting from a 
variety of assessment tools and 
focusing on different types of 
student achievement (i.e., at 

Tier III, II, and I) offer a more 
complete account of student 
knowledge just as these 
separate photos provide the 
"big picture" when grouped 

together. 
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• Some snapshots may reveal few specifics 
about the subject, but when grouped with 
accompanying photos, the "big picture" 
emerges. 



• Similarly, some assessment data, such as 
results from annual large-scale tests (Tier 
III), are limited in what they reveal about 
specific strengths and wealaiesses of a 
student, but when aggregated at the school 
or district level, they offer a "big picture" of 
student performance. 
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• Some snapshots are taken at 
close range in order to 
focus on specific details 
about the subject 




• Similarly, some assessment 
data, such as results from a 
teacher-administered math 
test (Tier I), focus on 
specific components of 
student knowledge. 
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Guiding Questions for Collecting Student Assessment Data 

• What evidence can we collect of our students’ achievement? 

• What types of assessment data can be collected over time so we can track student learning? 

• What knowledge and skills have our students achieved? 

• At what level of academic proficiency are our students? 

• To what degree are our students reaching academic standards? 

• How do our students compare to. . .?" (e.g., similar students at other schools) 

As data are identified that help answer these questions, it is important that: 

• The data are reliable — the data can be trusted and are stable and consistent across time. 

• The data are valid — the data measure what they were intended to measure. 

• The data are accessible. 

• Information about the data is accessible (from the test manuals, and so on). 

Three additional guiding questions to consider when collecting student assessment data are illustrated in the following 
graphs: 

1 . How are our students achieving as they progress through the grades? 
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2. Has the performance of our fifth graders improved within this school year? 



Year 1 





Fall 


Winter 


Spring 


Grade 

5 


Student 
Group A 




Student 
Group A 




Student 
Group A 



3. How are our fourth graders performing compared to fourth graders from previous years? 
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All three of these questions allude to longitudinal results and are also important to include because they may indicate 
patterns of changes across time. Longitudinal results of student assessment, collected and tracked over time, are essential 
to reveal trends in student learning. 

The first two of these three questions compare the performance of the same set of students across time, while the last 
question compares the performance of different sets of students across time. These two question types represent two very 
common methods (longitudinal and cohort analyses, respectively) used to indicate patterns of changes across time. 
However, both pose some considerations that improvement teams should be aware of when collecting and analyzing 
assessment data. When comparing the same set of students across different grade levels, it is important to remember that 
differing results of the same set of students across the years could simply be due to age or to any number of 
uncontrollable factors that have affected their learning. When comparing different sets of students across the same grade 
level, it is important to remember that differing results across time could simply reflect differences in their demographic 
makeup. 

Because of these considerations, it is important that both types of data are collected and that as these data are identified, 



the instruments, or assessment tools, used to collect the data meet the following criteria: 

1 . The data instruments are administered at the same time every year. 

2. The data instruments follow consistent, standardized, proper administration procedures. 

3. The data instruments contain parallel test items. 

4. The resulting scores from data instruments are provided on a consistent scale. 

A checklist of some suggested sources for Tier III, Tier II, and, Tier I assessment data is available in Appendix A. 

2. Student Data 

In an era of accountability and increasing school choice, schools must carefully track their communities and come to 
know them well. When determining which student data to collect, the goal is to thoroughly know the school population in 
order to clarify problems and needs. Student data includes both demographic data and behavioral data. Demographic data, 
such as a student's gender, ethnicity, or economic status, is relatively static and beyond the student's control. Behavioral 
data, such as student attendance or school suspensions, is dynamic and within the student's control. Data should be 
collected that show: 

• Who enrolls in the schools 

• Mobility patterns in and out of grades and schools 

• Neighborhood characteristics 

• Student transportation needs 

• Parent involvement 

• Rate of enrollments in special programs 

• Behavior and social problems of students. 

It is best to collect student data longitudinally over a five-year period so that trends can be viewed and predictions made. 
Data from previous years should be organized in a manner that will facilitate comparisons from year to year. 

Guiding Questions for Collecting Student Data 

• Who are our students? 

• What trends do we see in our student population? 

• What factors outside the school may help us understand our students? 

A checklist of student data that can be collected is available in Appendix B. 

3. Program Data 

Rich sources of information about the quality of programs in the school are often hidden and not collected. These data are 
not always readily quantifiable but are important and often telling in how they can support a hypothesis. Time should be 
taken with the leadership team to sort out the questions they have about their programs. In particular, data should be 
collected when there are questions about student success or student achievement. For example, data about the time 
demands of the instrumental music program may be important to collect if there is a question about band students' 
academic achievements. Programs can include a wide variety of offerings, from specially funded programs to academic 
curricular sequences to extracurricular programs. 

Plans should be made prior to the school year to collect program evaluation data. The collection of these data can be seen 
as "action research," which involves collecting data that will inform future decision making about programs and curricula. 
To prepare for an analysis of educational programs, collect data that profile the enrollment in your school's programs and 
courses. In addition to collecting information about student enrollments and performances, personnel should collect data 
about the implementation of standards-based curricula. 

Guiding Question for Collecting Program Data 

• How successful are our programs in bringing about the academic excellence articulated in our 
standards? 

A checklist of sources of program data is available in Appendix C. 

4. Perceptions Data 

The fourth type of data that are important to collect and evaluate are school community perceptions data. As a matter of 
practice, educators fail to pay attention to the members of the school community. These educators need to recognize that 
we have many different members in our school communities, and how they value our services impacts us profoundly. 
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To evaluate satisfaction, the leadership team should begin with a list of members that make up a school community: 

• Students 

• Parents 

• Teachers and staff 

• Community citizens 

• Community businesses 

• Regional colleges and universities 

• School board 

Guiding Questions for Collecting Perception Data 

• How do the members of our school community feel about our school and district? 

• How satisfied are school community members about our educational programs? 

• What do the members of our school community perceive to be the strengths and needs in our 
school? 

• What do the members of our school community think about the skills of our graduates? 

Data collection should be orchestrated to provide an honest portrayal of the district’s and school's climate. These data, 
often seen as intangible but clearly perceived by members of the school community, can be collected in creative ways. 
Surveys, polls, even analyses of local newspaper editorials and letters can suggest a school climate. If school community 
data are not readily available, the team should plan to collect perceptions data in their areas of need. 

A checklist of sources of perception data is available in Appendix D. 



Step 3: Analyze Data Patterns 



Assessment, student, program, and perception data are best analyzed through four lenses. It's a "quadnocular" view that, 
when brought together, shows clear patterns which help in developing improvement plans and strategies. 

The illustration below captures the essence of analyzing data through this quadnocular view. 



This analysis is deliberate and workable. Because the primary emphasis in school improvement is on student learning, 
analysis of assessment data is the first and foundational lens for all other data analyses. As team members sort through 
the other three lenses — student, program, and perception data — they continue their study using the assessment 
magnifying glass. 

The goal here is to uncover patterns and relationships among the data. While it is true that analyses can be conducted with 
statistical programs and electronic data tools, there is another process that cannot be overemphasized: digging through the 
data, finding patterns, diagramming observations, and collaborating about what is seen. It is a powerful process. We 
have seen individuals discover new ideas and views by collaborating with their teammates-discoveries they would never 
have made on their own. 

To begin uncovering patterns and relationships among the data, this tutorial recommends and demonstrates what has been 
termed the "stoplight method." This method is a "hands-on" approach to analyzing data. Leadership teams code raw data 
that is printed on paper. They use colored highlighters to indicate whether the data being assessed is below or above 
expectations. The stoplight method can be applied to any type of data (assessment, student, program, or perception). It 
can also be applied to any level of analysis-disaggregated (e.g., student level) or aggregated (e.g., school level) data. 
Leadership teams also illustrate and chart their own data. These methods help educators not only "see" more clearly, but 
also engage in their own professional growth with their own data. 

The sections below describe how the stoplight method can be used to analyze the four types of data. 



1. Analyzing Assessment Data 

Perhaps the most common source of assessment data is the state and district annual tests (Tier III data), which are 
important for the reasons listed below: 




• Public Accountability. District and especially state assessment results for individual schools and 
districts are reported publicly in newspapers and on the Web. These results are used in a variety 
of federal and state evaluation requirements. Educators ought to be "in the know" about these 
data. They should ask themselves, What are those scores? Where did they come from? What do 
they mean? Furthermore, most states require districts and schools to use these data to demonstrate 
"continuous improvement," according to processes unique to each state. 

• Evaluating Instruction and Curriculum. State and district assessments are designed to assess 
student performance in broad domains of academic achievement. They provide reliable indicators 
of student proficiency in core skill areas. Some educators might question their validity in terms of 
measuring local curricula. Upon close study, it is generally agreed that the items are valid 
measures of academic knowledge, skill, and understanding. Studying assessment results 
provides clues into the district's effectiveness in teaching core curricula. 

• Analyzing Trends. The data from standardized assessments tend to be reliable and can be viewed 
to observe trends over time, across subjects, and among grade levels. If the number of students is 
adequate, data can be reported by disaggregated groups of students to study how groups 
perform. 



Because of the importance of state and district annual assessments, this tutorial uses this source as a basis for analyzing 
assessment data. However, it is important to note that this basic sequence may be applied to any assessment data. In fact, 
we encourage you to apply this sequence to assessment data other than state and district annual tests (i.e., Tier II and Tier I 
assessment data). It is important to make decisions using multiple data sources because the decisions are more reliable when 
several sources of information suggest the same story. If the story varies based on the source of information, the measures 
from each source may not be evaluating the same concept. 

The sequence of analyzing assessment results described below proceeds systematically, from broad indicators, such as 
grade and subject, to more detailed results, such as test objective and test item. 

Analyzing Broad Results by Grade and Subject 

State and district annual assessments present results in what can be termed broad "proficiency summaries." These data 
indicate the percentage of students who met or exceeded the "proficient level." State and district annual assessments also 
present results in other ways, but the "proficiency summaries" provide the best analysis for measuring student performance 
toward academic standards, standards required by the nation and the state. Thus, the analysis process described in this 
tutorial is a "proficiency analysis." The table below indicates how each state in the NCREL region defines "proficiency." 



State Proficiency Targets 


State 


Data That May Be Used for Broad Analyses: 


Illinois 


IS AT: % of students at " meets or exceeds" standards 


Indiana 


ISTEP+: % of students " above the Indiana Standard" 


Iowa* 


ITBS: % of students at " intermediate and high levels " 


Minnesota 


BASIC STANDARDS: % of students 

COMPEHENSIVE ASSESSMENT: % of students at "Levels II, III, and IV” 


Michigan 


MEAP: % of students at "met or exceeded " levels 


Ohio 


OPT: % of students at " proficient and advanced" 


Wisconson 


WKCE: % of students at "proficient and advanced ' 
Writing: % of students at " 4.0 or above" 

WRCT: % of students at "basic or above " 



*For Iowa, the ITBS is used as an example. It is not a state-required assessment. 

The following section gives a step-by-step guide to conducting a broad analysis (by subject and grade) of state assessment 
data using the stoplight method. 

While again we emphasize the importance of digging through data to uncover patterns and relationships among the data, the 
Wisconsin Information Network for Successful Schools (WINSS) and the Illinois State Board of Education (ISBE) both 
have Web sites that provide many of these analyses for you once you specify your school or district in these states. The 
links to these Web sites are provided below: 
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WINSS ( http://www.dpi.state.wi.us/sig/index.htmn 
ISBE ( http://ilsi.isbe.net/ ) 



To conduct a broad analysis by subject and grade, use the following guiding question and steps: 
Guiding Question: How many of our students are performing at proficient or higher levels? 
Steps: 



1. Find your broad district and school data summaries from the state assessments. 

2. Locate the report that shows the percentage of students at each level (i.e., minimal, basic, 
proficient, and advanced). Focus on the percentages of all students enrolled (usually for a full 
academic year). Do not exclude special education or other exempted students, since the 
objective is to analyze data for all students enrolled. (Note that the levels may be listed as "scale 
score" ranges, so you would look for the percentage of students in each scale score range.) 

3. Calculate the percentage of all students at and above the standard level (i.e., at proficient and 
advanced) for each grade and subject. Use the guidelines for your state's assessment system. 
Create a chart like the one that follows, and only include the grade levels and subjects 
corresponding to the data that are reported in your state. 

4. If data are available for more than one year, create a chart that lists the data for the past three to 
five years. Having data from the past five years will allow your leadership team to compare 
how the same set of students perform across different grade levels and how different sets of 
students perform at the same grade level. 



Percentage of Students at and Levels 

(e.g., Meets and Exceeds) 


Grade 


School Year 
(e.g., 1999 to 2000) 


Reading 


Lang. 


Math 


Science 


Social 

Studies 


Writing 
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to 

to 

to 

to 
















to 

to 

to 

to 

to 
















to 

to 

to 

to 

to 
















to 

to 

to 

to 

to 















5. Using highlighters, highlight the numbers in the charts (small and large charts) using the color chart below. 
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Highlight 

Color 


Meaning 


% of Students 
(suggested cutoffs) 


% of Students 
(our cutoff levels) 


Blue 


WOW! Beyond Expectations 


90% and 
above 




Green 


GOOD! Meets 
Expectations 


80 to 89% 




Yellow 


CAUTION! Below 
Expectations 


70 to 79% 




Pink 


URGENT! In need of 
immediate improvement. 


0 to 69% 





The sample below shows a data chart with numbers highlighted. 



Grade 


School Year 


Rdng 


Lng 


Math 


A 


2000 to 01 


78% 


| 90% | 


70% 


4 


2001 to 02 


82% 


! 84% , 


71% 


Q 


2000 to 01 


60% 


60% 


49% 


O 


2001 to 02 


57% 


65% 


52% 




2000 to 01 


65% 


72% 


60% 


1U 


2001 to 02 


71% 


74% 


62% 



6. Note how patterns begin to emerge. 

7. Now, graph or visually represent the numbers indicated on the chart. Some data are better suited to certain types of 
graphs and charts than others. However, usually any visual representation (i.e., in a line or bar graph) is helpful. Graphs 
A, B, and C below provide some examples. As your team graphs more and more data, you will get a better sense of what 
graphs best display different types of data. 

Graphs A and B are examples of line graphs. Both graphs indicate the percentage of students at proficient and advanced 
levels by grade and subject for a single school year. Graph A indicates these results by representing the subject on the 
bottom axis and the grade level in the legend. The reverse is shown in Graph B. These graphs allow for comparisons to be 
made between different sets of students (different cohorts of students) at different grade levels for a single school year. 

Graph A 



best copy available 
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Percent of Students at Proficient and Advanced Levels by 
Grade and Subject (1998-1999) 




Graph B 

Percent of Students at Proficient and Advanced Levels by 
Grade and Subject (1998-1999) 

100 - 




Graph C is an example of a bar graph. It indicates the percentage of 4th- and 8th-grade students at proficient and advanced 
levels in a single subject (reading) for multiple school years (1996-97 through 2000-01). This graph is especially helpful 
because it allows for a longitudinal comparison of a single cohort of students from fourth to eighth grade. This cohort of 
students is represented in the graph with burgundy-colored bars. 
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Percent of Students at Proficient and Advanced Levels 
in Reading by Year and Grade 



90 * 




1996- 1997- 1998- 1999-2000- 1996- 1997- 1998- 1999- 2000- 
97 98 99 00 01 97 98 99 00 01 



4th Grade 8th Grade 

Year and Grade 



8. When you have finished creating the charts and graphs, consider the following questions: 

• What patterns do we see in this year's results? 

• Do we see similar patterns in past years' results? 

• What trends emerge over the past several years? Are these trends moving toward our goals? 

• Do these data surprise us? 

• Are there other broad data that show similar patterns? 

Analyzing Results by Objective 

Because the broad analysis of assessment only provides overall indicators by subject area, it is helpful to analyze results by 
objective. Doing this reveals more specifically how students perform on particular elements within a subject area strand. 
While analyzing results by objective, use the following guiding question: 

Guiding Question: How do our students perform on the knowledge, skills, and concepts tested in the broad content 
domains? 

Steps: The steps to analyzing assessment results by objective are similar to the steps listed in analyzing assessment by 
subject. One obvious difference is that instead of looking at a whole subject area, you will be looking at the objectives 
within those subject areas. The "test objective" report contained in the state assessment data may be called: 

• Objectives performance report 

• Subtest report 

• Skills report 

• Mastery objectives report 

• Other similar report titles 

These results will show how students performed within each of the content areas assessed. A second difference is that 
instead of applying the stoplight method to the percentage of students meeting or exceeding standards, you will be applying 
it to the average score for each objective. These scores should be next to the objective and may be called average percent 
correct scores, mastery scores, criterion scores, or objective performance index scores. What these scores are and what 
they mean should be found either on the front or the back of the report or in an accompanying administrative or 
interpretation manual. 

Because average percent score for all students is given, you must first find cutoff levels for the objective scores before you 
can highlight them according to the stoplight method. The purpose is to establish what is high performance on these 
objectives and what is low performance. The report itself may provide cutoff levels — such as low, average, or high 
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mastery — or you may have to set cutoff levels yourself. The first table below provides some sample objective score cutoffs. 
The second table is an example of how an objective data chart might look once cutoff levels are set and scores are 
highlighted. 



Highlight 

Color 


Meaning 


% of Students 
(suggested cutoffs) 


% of Students 
(our cutoff levels) 


Blue 


WOW! Beyond Expectations 


90% and 
above 




Green 


GOOD! Meets 
Expectations 


75 to 89% 




Yellow 


CAUTION! Below 
Expectations 


50 to 74% 




Pink 


URGENT! In need of 
immediate improvement. 


0 to 49% 





Content Area: Math Grade 8 


Objective 


School Year 


Average Percent Score 
for All Studemts 


Problem Solving 


1998 to 1999 

1999 to 2000 


51% 

58% 


Reasoning 


1998 to 1999 


49% 


1999 to 2000 


55% 


Geometry 


1998 to 1999 


61% 


1999 to 2000 


59% 


Numbers 


1998 to 1999 

1999 to 2000 


65% 

71% 


Algebra 


1998 to 1999 


48% 


1999 to 2000 


50% 


Measuring 


1998 to 1999 


70% 


1999 to 2000 


65% 


Statistics 


1998 to 1999 

1999 to 2000 


58% 

63% 



There are two caveats that must be mentioned when looking at average scores. First, an average score is an average; it tells 
nothing about the percentage of students who performed better than the average score or who performed worse. In this 
sense, the average score disguises the spread of student scores. For example, suppose the average percent score was 61 
percent. This disguises the 27 percent of students who "mastered" the objective (who scored 90 percent or above). 
Therefore, it is important to look not only at the average percent score for all students, but to look at the data that show the 
percentage of students who mastered each objective. It is also important to keep in mind that the average scores themselves 
should be interpreted with caution when the percentage of students tested is low in that the score provides no information 
about the achievement of students not tested. 

Second, each objective is tested with a number of items. This number of items may vary from objective to objective. 
Therefore, the average scores for content objectives cannot be directly compared to each other, and statements reflecting 
"better than" and "worse than" conclusions should be avoided. For example, if two of the content objectives for a state's 
reading area tested students' ability to (1) demonstrate a basic understanding of text and (2) to generate inferences, it would 
not be appropriate to make the conclusion, "Our students were worse at generating inferences than in demonstrating a basic 
understanding of the text." A better conclusion would be based on how well the students performed on each test objective 
based on the cutoff levels you set — "Our students performed below expectations on generating inferences" — or by 
comparing the average score of your school to that of the state or the nation. This caveat also applies to comparing average 
scores for each objective across years. This is because not only do the number of test questions measuring each objective 
change each year, but the test questions themselves change and are only roughly equivalent. 

As with the subject data charts, the objective data charts can be graphed to show yearly comparisons, so you can compare 
patterns observed by objective with results from the previous year. 

Analyzing Results by Individual Test Item 

Further details about student achievement can be gleaned from the study of item analyses, (the individual items on the test), 
if provided by the vendor. However, because of the number of precautionary measures involved with item analyses and 
because of the variation in types of items across individual states, it is beyond the scope of this tutorial to give a complete 



28 



coverage of them. Information on these precautionary measures and a demonstration on how to apply the stoplight method 
to an item analysis can be found in the Data Retreat Facilitator's Guide (coming soon). 

2. Analyzing Student Data 

Assessment results can also be used to help educators understand their students. The objective is to understand how 
individuals and groups of students perform to provide insight for developing fully equitable opportunities to learn. 

To analyze student data, use the following guiding questions and steps: 

Guiding Questions: 

• What does our data tell us about our students who are not at proficient levels? 

• What patterns do we see in our students who perform at or above proficient levels? 

• Since we cannot influence the variables regarding demographics, how can we be sure all of our 
students receive a full opportunity to learn? 

Steps: 

1 . Determine student variables. Decide which student characteristics, both demographic and 
behavioral, are unique to your team's students in light of their assessment results. Consider, for 
example, the following: 

o Distance from school 
o Gender 
o Mobility 
o Ethnicity 

o Behavior issues (referred for behavior X number of times) 
o Transfers 

o Family/personal crisis 
o Absenteeism 
o Disability 
o Truancy 

2. Highlight proficiency scores. Highlighting proficiency scores will proceed in one of two ways 
depending on what is provided in your state's assessment data. For some of these demographic 
variables, the proficiency scores will already be aggregated (summed or averaged) to the school 
level and will be provided in one of the reports. For example, there may be a report indicating the 
percentage of students at each proficiency level by ethnicity. 

If your state does provide aggregated reports for the demographic variables you are interested in, 
the highlighting process and analysis will be straightforward. Highlight the percentage of students 
at each proficiency level for each group or category (e.g., Caucasian, Hispanic, African American, 

American Indian, and Asian for the demographic variable of ethnicity) according to the stoplight 
method. 

If your state does not provide aggregated reports for the demographic variables you are interested 
in, the highlighting process will be more tedious but manageable. Locate the student data report 

and highlight each student's proficiency level according to the stoplight method.J- Then identify 
which students fall within the categories of the demographic variables you are interested in. You 
and your team may need to refer to other information sources (e.g., attendance reports if one of 
your chosen demographic variables was absenteeism) while coding the student data. The table 
below shows how a data report might look after your coding is completed. 

3. Discuss and display patterns in the data. Once your highlighting is completed, look for patterns in 
the data. At this point, do not concentrate on individual student names but focus on patterns among 
all of the students. Your team should then discuss these patterns and list observed strengths, 
weaknesses, and other unique patterns. For example, in the table above, there is an observed 
weakness for those students who have behavior problems. These students are the only ones who 
have a proficiency level of 1 in math. 





Reading 


Language 


Math 


Science 


Social 

Studies 


Student B 

Sports 








PL 2 




PL 4 | 


PL 4 1 


PL 3 j 


PIT3 | 
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Student C 

Absent 1 0+ Days 


PL 2 




PL 2 


PL 2 


PL 2 


Student D 

Behavior Problems 


" P L 3 “ 


PL 2 


PL 1 


PL 2 


PL 1 


Student E 

Behavior Problems 


PL 2 


PL 1 


PL 1 


; pl 3 • 


PL 2 


PL=Performance Level 



3. Analyzing Program Implementation Data 

The goal in analyzing program data is to discover any patterns that may exist regarding the implementation of certain 
programs in the school. The analysis of the program data proceeds similarly to the analysis of the demographic variables 
that are not aggregated in the state assessment data. 

Guiding Questions: 

• What program participation or enrollment information should we study in light of assessment results? 

• What program patterns do we see among our students who perform below proficient? 

Steps: 

1. Determine program variables. Brainstorm all of the possible program implementation variables 
that would be unique to your team's students. For example, you may want to consider 
enrollment or participation in: 

o Gifted and talented program 
o Title 1 program 
o Athletics 

o Extracurricular activities 
o Preschool 

o Alternative high school 
o After-school employment 
o Other 

2. Highlight proficiency scores. Most likely, your team may have already chosen some student 
variables in which the state did not provide an aggregate report. If so, you will have already 
highlighted student scores according to each student’s proficiency level. 

3. Discuss and display patterns in the data. Identify which students fall under the program 
implementation categories your team selected and look for patterns, identifying strengths and 
weaknesses, just as your team did for the student variables. Again, it is important not to 
concentrate on individual student names but instead focus on patterns among all of the students. 



4. Analyzing Perception Data 

Many patterns will have emerged through the in-depth look at assessment, student, and program data patterns. The last area 
of study-perceptions analysis-is designed to examine the patterns that exist among the perceptions of the school district's 
community. 

To analyze perceptions data, use the following guiding questions and steps: 

Guiding Questions: 

• What do the members of our school community think about the job we’re doing? 

• If we were doing our job, and the members of our school community were very satisfied with 
our work, what would the data look like? 

• What evidence do we have that our students are satisfied with our job? 

• What evidence do we have that our parents are satisfied? 

• What evidence do we have that our staff is satisfied? 

• What evidence do we have that our community is satisfied? 
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1. Determine expectations. Answering the guiding questions above will help your team determine your team's expectations. 
You are in a position to look for patterns in the data according to those expectations. For example, if your school 
implemented activities to increase parental involvement, your team should expect results that reflect those changes. 

2. Set survey guidelines. Perceptions data will take unique formats depending on the nature of the data, but what is 
important is to look for patterns according to the team’s expectations. Based on those expectations, your team should assign 
stoplight values just as was done with the previous sets of data. 

As an example, consider a 30-item climate survey administered to parents, students, and teachers. Your team might 
determine positive results according to the percentage of rankings of three or higher on a five-point satisfaction scale. Thus, 
using this example, your team may decide that if 85-100 percent of the rankings were three or higher, the color coded 
should be blue; if 70-84 percent of the rankings were three or higher, the color coded should be green; and so on (see table 
below). 



Highlight 

Color 


Meaning 


Survey Results 
(% of ranking of 3 or higher on a 
5-point scale) 


Our 

Criteria 


Blue 


WOW! 
ADVANCED 
Beyond Expectations 


85 to 100% 




Green 


GOOD! 
PROFICIENT 
Meets Expectations 


70 to 84% 




Yellow 


CAUTION! 

BASIC 

Below Expectations 


55 to 69% 




Pink 


URGENT! 

MINIMAL 

In need of immediate 
improvement 


0 to 54% 





3. Discuss and display patterns in the data. As survey data are shared and studied, team members should summarize and 
articulate patterns. Important patterns are those among members of the school community-ffom parents, to children, to 
school staff members. Major problem areas as well as demonstrated strengths should be listed. 



Summarize AH Problems and Strengths 

Once you have analyzed your team's assessment, student, program, and perceptions data, it is time to pull all of the 
observations together — to move from looking at details to stepping back and looking at it all from a distance. This step is 
the transition from analysis to interpretation. 

To do this, your team must summarize observed strengths, and summarize and rank observed problems across all data. To 
summarize all problems and strengths, use the following guiding question and steps: 

Guiding question: Based on all the data we have studied and the patterns we have observed, what is the sum of 
problems that have emerged from the data? 

Steps: 



1 . Determine group process for making decisions. Making these decisions is easiest with either a 
democratic or consensus process. The team should decide the process they will use. 

2. List and celebrate strengths. Team members, using the summaries of all data analyses-from 
assessment to students to program to perception-should discuss and list all observed strengths. 
The team should then celebrate and acknowledge all of the positive accomplishments revealed 
through the data. The team should also consider how these strengths may be shared (e.g., 
newsletters and radio programs). 

3. List and rank order problems. Using the summaries of all data analyses-from assessment, to 
students, to program, to perception-team members should discuss and list all observed 
problems. Focus on the problems that can be changed through intervention. With these 
problems listed, the team should rank order them so that the problems proceed in order of 
urgency. 
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BEST COPY AVAILABLE 



Endnote 

1 . During this segment of data, information may be shared about specific students. If this is the case, the members of the 
leadership team need to respect the confidentiality of this specific student information. 
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Step 4: Pose Hypotheses 



Formulating questions in response to the data (e.g., Why are our eighth-grade students meeting the standards in math but 
not in language arts?) and considering responses to these questions, often by consulting additional data, may lead to 
possible explanations for observed data patterns. These are called hypotheses. The goal of this process in terms of the 
data presented here is to try and get closer to the root causes of our children's performance problems. This enables us to 
take specific actions to help our children perform at the levels of excellence set forth. The posing of hypotheses can be 
encouraged and recorded during the data analyses phase but should also receive special attention after the data patterns are 
sorted. 

During this phase, the team should use the following guiding questions and steps: 

Guiding Questions: 

• Why are our children performing the way they are? 

• What in our systems and practices is causing our children to have these problems? 



Steps: 

1. Set team ground rules. Since team members will have their own ideas about why things are the way they are, ground 
rules should be determined and enforced. The following are some suggested ground rules for teams to adhere to while 
generating and refining hypotheses: 



Team Ground Rules 

1 . Designate one team member as the recorder who lists the hypotheses as they are generated. 

2. Encourage every team member to contribute hypotheses. 

3. Respectfully listen to each member's hypothesis. 

4. List all hypotheses without rejection or editorial comment. 

5. Hypotheses that are challenged are discussed professionally with evidence and reasoning to dispute or 
accept them. 



2. Record and accept or reject hypotheses. For each problem statement, have a team member write ideas on a chart. As 
these hypotheses are generated and listed, label them as accepted or rejected and indicate the reasons for doing so. The chart 
below lists both accepted and rejected hypotheses for a sample problem statement. 
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Sample Hypothesis Testing 



Problem : Achievement levels in math drop grade by grade until they are at very low levels in Grades 6, 7, 
and 8. They pick up only slightly from Grades 9 to 12. 



Sample Hypothesis 


Evidence to the Contrary? 


Is it because there are more special education 
students each year in regular classes and it pulls 
our scores down? 


REJECT. We checked special ed enrollments. They do 
increase from Grades K-3, but then stabilize until 
Grade 6, and then decline to Grade 12. 


Our standards are just too high. The tests are just 
too difficult, year by year. 


REJECT. We looked at test results nationally and in 
neighboring districts. Although mathematics 
performance is low nationally and statewide, our 
performance is particularly low compared to our 
neighbors and to the national sample. We have also 
studied the items and concur that the items are fair for 
the grade levels assessed. 


Our math teachers in the intermediate and middle 
levels have not had the proper training to teach the 
current math standards. 


ACCEPT AS A POSSIBILITY. We looked at the 
licenses, and the teachers do have appropriate 
credentials. However, we looked at the sequence and 
record of professional development activities, and our 
district has provided no math professional development 
in 10 years. 


Our textbooks are not only out of date, but were 
not adopted in a logical grade-by-grade sequence. 


ACCEPT AS A POSSIBILITY. We charted our math 
textbook adoptions. They range from 1981 to 1986 
from five different publishers. We are long overdue for 
new materials adoption. 


Our students are apathetic. They are turned off and 
just don't care enough to do their homework. 


REJECT. We checked attendance rates and behavior 
problems, and see no real pattern there with math 
performance. Many of these same students perform 
well in other subjects. 



3. Keep the focus sharp. Members should spend the time to focus as specifically as possible on "causes" of problem. For 
example, it's not enough to say, "We have low scores because our students have trouble with multistep tasks." Go a step 
further and ask each other, "Why are our children having trouble with multistep tasks?" 



Step 5: Develop Improvement Goals 

Now that data patterns have been analyzed, problem areas prioritized, and hypotheses generated, your team is ready to 
develop goals for improvement. Your team should work both on long-range goals (five years from now) and on 
short-range goals (those that are to be achieved within one school year). The first step is to focus on the most urgent 
problem and its hypothesis. Considering that problem, your team should use the following guiding question and steps: 

Guiding Question: What outcome of improvement will we set for our students regarding this problem? 

Steps: 



1 . Discuss the outcome you want for your students five years from now. 

2. Project one year toward that goal. What outcome will you set for yourselves to attain within a 
year? 

3. Think about the capacities in your staff and your students and the barriers that must be 
overcome. 

4. Discuss the level of commitment (e.g., time, finances, and so on) it will take to reach this 
outcome. During this discussion, avoid talking about specific strategies and instead focus on 
the goal. 

5. Discuss what your data will look like a year from now when you've achieved this goal. 

The second step is to develop a one-year goal statement about your most urgent problem. Your team should remain focused 
on this goal until consensus is reached about the exact wording of the goal. The next two charts provide guidelines for 
developing goals and some sample improvement goal statements. 



Guidelines for Developing Goals 

Goals should be: 

• Clear. Goals should be focused and clearly stated. 

• Data based. The goals should be directly based on the observed patterns seen through the data and 
their connection to the evaluation criteria. 

• Few. Goals should be few in number; they should be substantive and focus on the primary purpose 
of improving student achievement. For example, a district may set two or three improvement goals. 

• Measurable. Goals should be measurable. They should articulate the desired outcome, not the 
specific strategies. (The hypotheses will help develop the specific strategies or actions to fulfill the 
goals addressed in Step 6.) 

• Sustainable. Goals should be systemic and sustainable. The goals should lead to system changes and 
adjustments that can be sustained into the future. 

• Community driven. Goals should be developed with outcomes that will meet the needs of the 
district's community-students, parents, school board members. 

• Developed by consensus. All team members should agree on all of the district goals. 

• Attainable. The goal should be one that can be achieved. Avoid unrealistic goals and aim for tangible, 
realistic goals that cause stretching but are attainable. 
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Sample Improvement Goals 

(Note that the "so that" phrase makes the goal visible and measurable) 

• To improve the mathematics performance of students in Grades 6, 7, and 8 so that at least 65 percent 
of the students in Grade 8 are at the proficient level, and 70 percent of the students in Grades 6 and 7 
surpass the Terra Nova median scale scores. 

• To improve student attendance so that 99 percent of our students attend school on 99 percent of 
school days. 

• To improve the reading performance of children in Grades 1, 2, and 3 so that 100 percent of the 
students in Grade 1 are reading at level 16;1 00 percent of the students in Grade 2 are reading at level 
30; and 100 percent of the students in Grade 3 reach the basic level or above on the WRCT. 

• To improve at-home support for school assignments and activities so that 100 percent of the students 
complete 100 percent of their assignments, and 100 percent of the parents participate in two 
conferences per year. 

• To improve the science performance of students in high school, Grades 9 and 10, so that 80 percent 
of the tenth-grade students perform at proficient or advanced levels, and so that there is an increase in 
student enrollment in elective science courses in Grades 1 1 and 12. 

• To improve the mathematics performance of students in Grades K-8 so that by the end of eighth 
grade, 80 percent of students are proficient in basic algebra and geometry. 

Once your team has developed a goal statement for your most urgent problem, move on to the next problem statement. Keep 
drafting each goal statement until your team has articulated a focused set of goals (e.g.., three to five goals). 
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Step 6: Design Specific Strategies 



Goals are meaningless unless action backs up the commitment. This part of the improvement planning process moves the 
hypotheses set forth by the team forward. Time must be allowed to do a careful, thorough job when designing these 
strategies. 

When clear goals are developed and are listed as top priorities, team members should begin to think about research or 
information they may have regarding that issue. For example, if there is a goal regarding improvement in writing, then 
team members should bring their own materials regarding that subject to the Data Retreat to use as a reference in 
designing strategies. 

To define a strategy for a particular goal, use the following guiding question and steps: 

Guiding Question: What specific actions will we take to achieve this improvement goal? 

Steps: 

1. Brainstorm. Your team must focus on the actions you can take to turn student performance 
around to meet a particular goal. While brainstorming strategies, think action. What specific 
actions will we take to achieve our improvement goals? What specifically can we do in our 
schools to make a real, measurable difference for our students? 

2. Use the hypotheses. Specific strategies can come naturally from the hypotheses that were 
accepted as possibilities. 

3. Design several strategies. There is a much better chance of reaching a goal when multiple related 
strategies are implemented throughout the entire school year. 

Below are (1) some additional guiding questions to consider when defining strategies and (2) two examples of defined 
strategies. These example strategies are based on the accepted hypotheses listed in Step 4. 



Guiding Questions for Strategies 

Is this strategy: 

1 . Clear and understandable to all readers and users? 

2. One specific action or activity? 

3. Dependent on other activities? (If so, be sure to describe the sequence of actions) 

4. An activity that will definitely lead to accomplishing the goal? 

5. Observable and measurable? 

6. Assignable to specific persons? 

7. Based on best practices ? 

8. One that all team members endorse? 

9. An action that will make a positive difference? 

10. Doable — one that can be implemented? 
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researching best practices. Your team can build-in a systematic process to investigate what other successful schools have 
done to meet a similar problem. The caution here is to conduct the research as quickly as possible so that subsequent actions 
can be added to the plan. 

It is important for the team to realize that strategies mean hard work. They are commitments to carrying out real action. 
Therefore, the team should take time to discuss the level of commitment and hard work necessary to carry them out. 

Some further considerations your team may want to include in your strategies are timelines (detailed dates and times the 
strategy is to be implemented), assigning duties (listing a person to be responsible), and documenting the plan (including 
them on the improvement plan for each goal). 
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Step 7: Define Evaluation Criteria 



At the close of a school year and in preparation for another year's data analysis, the team should be prepared to evaluate 
the success of its improvement efforts. Clearly defining the criteria at the beginning of the process will be greatly 
appreciated as you approach the end of the school year in the spring. 

If there is any area of goal setting that gets "short shrift," it's building in an evaluation plan from the start. It's one thing 
to set goals; it's quite another to deliberately evaluate your success, using data as your guide, against the initial goal. 

To develop an evaluation plan for specific strategies, team members should lay out the measures that will be used to 
examine how successful each strategy was. They should ask themselves the following guiding questions: 

Guiding Questions: 

• How will we know if our strategies were successful? 

• What evidence will we have to show the success of our action? 

• Did the strategies work and how will we know? 

Data that show the success of the various strategies and the degree of implementation are equally important to study along 
with data about achievement of the goal. It is important that the team stay focused on the desired measurable outcome and 
the evidence needed to show success. Some evaluation criteria may consist of: 

• Test scores 

• Attendance counts 

• Records of meetings held and actions accomplished 

• Observations 

• Survey tabulation 



All the measures that evaluate success of the strategies will imply that the goal was met. It is important, however, to set out a 
specific measure of the goal. If team members have written their goals to be measurable (as in the examples in Step 5), 
evaluation will be simple. 

Straightforward collection of the assessment or other specified data is necessary to evaluate whether or not the goal was met. 
Look back at the improvement goal, look for information about the data that will be collected, and determine what levels or 
criteria in the data will show fulfillment of the goal. 
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Step 8: Make the Commitment 



The final step is ensuring a commitment to the school improvement plan. Team members and responsible parties should 
sign the improvement plan, their agreement to work toward fulfilling the strategies clearly outlined. Signing a piece of 
paper at a leadership meeting, however, may easily be forgotten once the school year begins. To help solidify their 
commitment, teams can simply add their own commitment statement to each improvement plan. 

Below is an example of a written commitment. 



I understand and will fulfill my responsibilities for carrying out the 
strategies described in the improvement plan. 



Name, 



Date 



At one of the leadership meetings, teams should allocate time and devote it to their plan of "rollout" to the rest of the staff. 
It is important for the team to recognize the absence of the bulk of their staff. The rollout should be designed to: 



1 . Inform teachers about the data so they are aware of their school's challenges and celebrations. 

2. Cue teachers in on the patterns that exist in the data and share the list of observed problem areas 
in their ranked order. 

3. Summarize the various hypotheses that were posed. 

4. Share the full improvement plan. Sometimes it is a good idea to leave several blanks for 
strategies so that teachers in fall meeting sessions can add their own brainstorm ideas. This 
process helps to build teacher ownership of the plan of strategies. 

5. Assign Roles. All those who were on the team may want to think about their role in the rollout. 
In some schools these members actually split up the task of sharing the data and the plans in a 

. creative "back-to-school" challenge and kick-off. This plan works best if the team is composed 
of just as many teachers as administrators. 

6. Communicate the plan. Prepare something that describes the improvement plan clearly to all 
staff in writing (a special bulletin, newsletter or other communication means). Teachers can 
take it back to their classrooms and keep it next to their lesson plan books. Remind all staff that 
this is a whole staff commitment for the entire year. 
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The Value of Using Data Often 



The eight-step analysis process we promote in this tutorial to incorporate data into school improvement planning focused 
heavily on data that provides feedback on an annual basis. As we demonstrated, analyzing this data can be an insightful 
process in identifying strengths and weaknesses of students. Data collected annually can contribute to judgments made 
about the acquired skills of students, the effectiveness of teachers, and the quality of the curriculum after learning or 
instruction has taken place. 

If our goal is to improve the system, it cannot be emphasized enough that we need to gather data that tell us about the 
variables in the system on a continual basis. When it comes to working with assessment data, we do not want to make 
important programmatic decisions without a full view of how our students are learning. Yearly standardized tests are not 
adequate for a comprehensive assessment system. When analyzing the success of learning in the system, we need to look 
at progress to see if our students are improving and showing growth. Therefore, the state assessment data must be 
complemented with other tiers of data to make sound instructional and programmatic decisions. 

Building Data Literacy 

Developing literacy around the use of data to make decisions is valuable and has a major impact on a school's 
improvement effort. A long-range in-service training plan should be devised that could consist of several components: 

1 . Partnerships with area colleges and universities to devise hands-on learning experiences on using data. 

2. Forums and team meetings during the school year with guided assistance; teachers can meet in teams to 
review periodic and classroom assessment data. There is no better way to learn than to delve in with 
relevant, current data about kids in the classroom. 

3. A system in which teachers can serve as mentors to their peers, or mentors from a service agency or 
college/university can partner with teachers around the use of data. 

4. Attendance at guided Data Retreats. 

5. Periodic sessions throughout the year in which the leadership team comes together to go over ongoing 
data collected. Team membership should be on a rotating basis so all teachers have an opportunity to 
share. 

6. Administrators can require the use of data in supervision evaluation systems. Such requirements, 
however, should be supported with training, mentoring, and group work. 
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Appendix A 



Checklist for Tier III, Tier II, and Tier I Assessment Data 

Tier III Assessment Data 

□ State Assessment Data. Each of the State Assessments listed below should include the bulleted items: 





• State Assessment Administrator’s 


State Assessments 


Interpretation Guide 


Illinois: ISAT 


• State Assessment District and School 




Proficiency Summaries 


Indiana: ISTEP+, GQE 


• State Assessment Results by Objective 


Iowa: none required 


(or subtest) 




• State Assessment Results by 


Michigan: MEAP, HSPT 


Individual Test Items 


Minnesota:Basic Standards, Comp Assessment 


• State Assessment Disaggregated Data 


Ohio: Proficiency Tests 


• State Assessment Results by 




Individual Students 


Wisconsin: WKCE, WRCT 





U Standardized assessment results (e.g., Terra Nova, ITBS, ACT, SAT - all reports, including school and district 
summaries, objectives results, student rosters) 

□ Results from annual district grade level (benchmark) assessment 



Tier II Assessment Data 

□ Results from periodic district grade-level (benchmark) assessment 

□ Results from knowledge "probes" (e.g., curriculum-based measurement, computer assisted assessments, etc.) 

□ Results from district "end-of-course" exams 

□ Results from district-created assessments (e.g., criterion-referenced tests, writing assessments, performance 
assessments) 

□ Report card grades (with criteria); Ds and Fs lists 

□ Results from district-adopted writing assessments 

□ District rubric proficiency data on standards 
Tier I Assessment Data 

□ Results from primary-grade literacy assessments (running records, guided reading level achievements, etc.) 

□ Number correct and number incorrect on timings of basic skills (i.e., for fluency and automaticity). These results can 
be charted on a daily basis to see immediate changes in knowledge (also known as learning!). See Teachers and Students 



an example. 



O Results from student writing samples, scored according to rubrics. 

Q See http://ctl.uvm.edu/cat.html for other examples of classroom assessment techniques. 

Miscellaneous Sources of Assessment Data 

O IEP assessment data (especially students exempted from standardized assessment) 

Q Alternate assessment data for students with Limited English Proficiency 
O Preschool developmental progress data 
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Appendix B 

Checklist of Student Data 

Demographic 

Year of enrollment data (e.g., 1994, 1995, 2000, etc.) 

Details of transfer, including date and name of transferring school district (e.g., 1999, Fairview School District) 
Mode of transportation 

Students with special needs (e.g., disability, gifted, etc.) 

Students with limited English proficiency (data on their native language should also be included) 

Students who receive free and reduced lunches 
Migrant status 
Mobility 
Homelessness 
School-age parents 
Gender 
Ethnicity 

Parent education level 

Teen pregnancy 

Behavioral 

O Attendance 
O Truancy 

n Suspension and expulsion 
□ Behavior/social problems (referrals) 

O Student employment 
O Preschool attendance 
O Date of transfer out or dropout 

D 




□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 
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Appendix C 



Checklist of Program Data 



Q Course enrollments-course sequences 
Q Curriculum implementation data 

Q Alternative program data-enrollments, in alternative programs, such as alternative high schools or other special 
programs at alternative sites 

Q Work-based learning program data 

Q Graduation rates 

Q Postgraduation data-college enrollments, work/career information from follow-up student surveys 
Q Preschool program data 

Q State school report card data (number of advanced placement courses, attendance rates, etc.) 

Q Enrichment program information 

□ Special education program information 

D Extracurricular and cocurricular participation (i.e., athletics, clubs, community service) 

O Student/teacher ratio 
O Number of instructional aides 
n Parent/community volunteer data 
n Teacher credentials data 
Q Teacher attendance data 
n Teacher licensure data 

Q Data regarding teacher participation in professional development 

□ Data about the implementation of textbooks and other resources 
Q Number and types of field trips 

Q Strategic plan information 
Q Other student program and implementation data 
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Appendix D 



Checklist of Perception Data 



□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 



Parent surveys* 

Student surveys* 

School safety data 
Student wellness data 
Student self-concept data 
Equity survey data 
School climate data* 

Review of newspaper editorials and letters 
Hotline information (from a school call-in line) 
Suggestion box information 



*Some references where sample surveys can be found include: 

Bernhardt, V. L. (1999). The school portfolio: A comprehensive framework for school improvement (2nd ed.). 
Larchmont, NY: Eye on Education. 

Bernhardt, V. L. (1998). Data analysis for comprehensive schoolwide improvement. Larchmont, NY: Eye on Education. 

Sargent, J. W., & Smejkal, A. E. (2000). Targets for teachers: A self study guide for teachers in the age of standards. 
Winnipeg: Portage & Main Press. 
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